ELAIS-N2 master catalogue

Checks and diagnostics

from herschelhelp_internal import git_version
print("This notebook was run with herschelhelp_internal version: \n{}".format(git_version()))

This notebook was run with herschelhelp_internal version: 
04829ed (Thu Nov 2 16:57:19 2017 +0000)

%matplotlib inline
#%config InlineBackend.figure_format = 'svg'

import matplotlib.pyplot as plt
plt.rc('figure', figsize=(10, 6))
plt.style.use('ggplot')

import locale
locale.setlocale(locale.LC_ALL, 'en_GB')

import os
import time
import itertools

from astropy.coordinates import SkyCoord
from astropy.table import Table
from astropy import units as u
from astropy import visualization as vis
import numpy as np
from matplotlib_venn import venn2

from herschelhelp_internal.masterlist import nb_compare_mags, nb_ccplots, nb_histograms, find_last_ml_suffix

OUT_DIR = os.environ.get('OUT_DIR', "./data")
SUFFIX = find_last_ml_suffix()
#SUFFIX = "20170710"

master_catalogue_filename = "master_catalogue_elais-n2_{}.fits".format(SUFFIX)
master_catalogue = Table.read("{}/{}".format(OUT_DIR, master_catalogue_filename))

print("Diagnostics done using: {}".format(master_catalogue_filename))

Diagnostics done using: master_catalogue_elais-n2_20170720.fits

I - Summary of wavelength domains

flag_obs = master_catalogue['flag_optnir_obs']
flag_det = master_catalogue['flag_optnir_det']

venn2(
    [
        np.sum(flag_obs == 1),
        np.sum(flag_obs == 4),
        np.sum(flag_obs == 5)
    ],
    set_labels=('Optical', 'mid-IR'),
    subset_label_formatter=lambda x: "{}%".format(int(100*x/len(flag_obs)))
)
plt.title("Wavelength domain observations");

venn2(
    [
        np.sum(flag_det[flag_obs == 5] == 1),
        np.sum(flag_det[flag_obs == 5] == 4),
        np.sum(flag_det[flag_obs == 5] == 5)
    ],
    set_labels=('Optical', 'Mid-IR'),
    subset_label_formatter=lambda x: "{}%".format(int(100*x/np.sum(flag_det != 0)))
)
plt.title("Detection of the {} sources detected\n in any wavelength domains "
          "(among {} sources)".format(
              locale.format('%d', np.sum(flag_det != 0), grouping=True),
              locale.format('%d', len(flag_det), grouping=True)));

II - Comparing magnitudes in similar filters

The master list if composed of several catalogues containing magnitudes in similar filters on different instruments. We are comparing the magnitudes in these corresponding filters.

u_bands = ["WFC u", "CFHT Megacam u"]
g_bands = ["WFC g", "CFHT Megacam g", "GPC1 g", "RCS g"]
r_bands = ["WFC r", "CFHT Megacam r", "GPC1 r", "RCS r"]
i_bands = ["WFC i",                   "GPC1 i", "RCS i"]
z_bands = ["WFC z", "CFHT Megacam z", "GPC1 z", "RCS z"]
y_bands = [                           "GPC1 y", "RCS y"]

II.a - Comparing depths

We compare the histograms of the total aperture magnitudes of similar bands.

for bands in [u_bands, g_bands, r_bands, i_bands, z_bands, y_bands]:
    colnames = ["m_{}".format(band.replace(" ", "_").lower()) for band in bands]
    nb_histograms(master_catalogue, colnames, bands)

HELP warning: the column m_rcs_y (RCS y) is empty.

II.b - Comparing magnitudes

We compare one to one each magnitude in similar bands.

for band_of_a_kind in [u_bands, g_bands, r_bands, i_bands, z_bands, y_bands]:
    for band1, band2 in itertools.combinations(band_of_a_kind, 2):
        
        basecol1, basecol2 = band1.replace(" ", "_").lower(), band2.replace(" ", "_").lower()
        
        col1, col2 = "m_ap_{}".format(basecol1), "m_ap_{}".format(basecol2)
        nb_compare_mags(master_catalogue[col1], master_catalogue[col2], 
                        labels=("{} (aperture)".format(band1), "{} (aperture)".format(band2)))
        
        col1, col2 = "m_{}".format(basecol1), "m_{}".format(basecol2)
        nb_compare_mags(master_catalogue[col1], master_catalogue[col2], 
                        labels=("{} (total)".format(band1), "{} (total)".format(band2)))

CFHT Megacam u (aperture) - WFC u (aperture):
- Median: -0.04
- Median Absolute Deviation: 0.25
- 1% percentile: -0.8018979835510254
- 99% percentile: 3.7088455772399915

CFHT Megacam u (total) - WFC u (total):
- Median: -0.24
- Median Absolute Deviation: 0.20
- 1% percentile: -1.133989028930664
- 99% percentile: 2.9416127777099588

CFHT Megacam g (aperture) - WFC g (aperture):
- Median: 0.07
- Median Absolute Deviation: 0.16
- 1% percentile: -0.6488978195190429
- 99% percentile: 1.2106095886230488

CFHT Megacam g (total) - WFC g (total):
- Median: -0.06
- Median Absolute Deviation: 0.15
- 1% percentile: -1.013030776977539
- 99% percentile: 2.029975814819333

GPC1 g (aperture) - WFC g (aperture):
- Median: -0.04
- Median Absolute Deviation: 0.21
- 1% percentile: -1.409198970794678
- 99% percentile: 2.3876510620117206

GPC1 g (total) - WFC g (total):
- Median: 0.02
- Median Absolute Deviation: 0.14
- 1% percentile: -1.259632263183594
- 99% percentile: 0.8684431457519534

No sources have both WFC g (aperture) and RCS g (aperture) values.
RCS g (total) - WFC g (total):
- Median: -0.09
- Median Absolute Deviation: 0.17
- 1% percentile: -1.1273002624511719
- 99% percentile: 1.1714000701904297

GPC1 g (aperture) - CFHT Megacam g (aperture):
- Median: -0.10
- Median Absolute Deviation: 0.30
- 1% percentile: -3.1742844581604004
- 99% percentile: 2.404479503631592

GPC1 g (total) - CFHT Megacam g (total):
- Median: 0.09
- Median Absolute Deviation: 0.14
- 1% percentile: -3.2294483184814453
- 99% percentile: 0.9065727233886673

No sources have both CFHT Megacam g (aperture) and RCS g (aperture) values.
RCS g (total) - CFHT Megacam g (total):
- Median: -0.03
- Median Absolute Deviation: 0.15
- 1% percentile: -1.1794805526733398
- 99% percentile: 1.0631583595275873

No sources have both GPC1 g (aperture) and RCS g (aperture) values.
RCS g (total) - GPC1 g (total):
- Median: -0.14
- Median Absolute Deviation: 0.14
- 1% percentile: -0.9783744812011719
- 99% percentile: 1.1184496879577637

CFHT Megacam r (aperture) - WFC r (aperture):
- Median: 0.06
- Median Absolute Deviation: 0.15
- 1% percentile: -0.6783035278320313
- 99% percentile: 1.7159394073486296

CFHT Megacam r (total) - WFC r (total):
- Median: -0.10
- Median Absolute Deviation: 0.14
- 1% percentile: -1.0879096031188964
- 99% percentile: 2.0215468215942365

GPC1 r (aperture) - WFC r (aperture):
- Median: -0.05
- Median Absolute Deviation: 0.12
- 1% percentile: -0.9961013793945312
- 99% percentile: 1.3988991737365724

GPC1 r (total) - WFC r (total):
- Median: 0.01
- Median Absolute Deviation: 0.11
- 1% percentile: -0.9179030799865723
- 99% percentile: 0.7981997108459482

No sources have both WFC r (aperture) and RCS r (aperture) values.
RCS r (total) - WFC r (total):
- Median: -0.14
- Median Absolute Deviation: 0.14
- 1% percentile: -1.1533707427978517
- 99% percentile: 1.0309009552001953

GPC1 r (aperture) - CFHT Megacam r (aperture):
- Median: -0.11
- Median Absolute Deviation: 0.21
- 1% percentile: -3.751292018890381
- 99% percentile: 1.4144266510009773

GPC1 r (total) - CFHT Megacam r (total):
- Median: 0.11
- Median Absolute Deviation: 0.10
- 1% percentile: -2.8896341800689695
- 99% percentile: 0.8206905746459963

No sources have both CFHT Megacam r (aperture) and RCS r (aperture) values.
RCS r (total) - CFHT Megacam r (total):
- Median: -0.05
- Median Absolute Deviation: 0.15
- 1% percentile: -1.153233814239502
- 99% percentile: 1.0873112678527819

No sources have both GPC1 r (aperture) and RCS r (aperture) values.
RCS r (total) - GPC1 r (total):
- Median: -0.16
- Median Absolute Deviation: 0.10
- 1% percentile: -0.8823718261718749
- 99% percentile: 0.626745071411132

GPC1 i (aperture) - WFC i (aperture):
- Median: 0.00
- Median Absolute Deviation: 0.11
- 1% percentile: -0.8453998565673828
- 99% percentile: 0.9095308685302732

GPC1 i (total) - WFC i (total):
- Median: 0.07
- Median Absolute Deviation: 0.11
- 1% percentile: -0.7359315490722655
- 99% percentile: 0.8652436828613284

No sources have both WFC i (aperture) and RCS i (aperture) values.
RCS i (total) - WFC i (total):
- Median: -0.10
- Median Absolute Deviation: 0.18
- 1% percentile: -1.1100502014160156
- 99% percentile: 1.3543500900268555

No sources have both GPC1 i (aperture) and RCS i (aperture) values.
RCS i (total) - GPC1 i (total):
- Median: -0.20
- Median Absolute Deviation: 0.08
- 1% percentile: -0.8265383529663086
- 99% percentile: 0.4684374618530278

CFHT Megacam z (aperture) - WFC z (aperture):
- Median: 0.14
- Median Absolute Deviation: 0.17
- 1% percentile: -0.6702371978759765
- 99% percentile: 3.5893981742858516

CFHT Megacam z (total) - WFC z (total):
- Median: -0.04
- Median Absolute Deviation: 0.20
- 1% percentile: -0.9459419250488281
- 99% percentile: 2.56119647979736

GPC1 z (aperture) - WFC z (aperture):
- Median: 0.03
- Median Absolute Deviation: 0.12
- 1% percentile: -0.8302993774414062
- 99% percentile: 1.1431929779052727

GPC1 z (total) - WFC z (total):
- Median: 0.08
- Median Absolute Deviation: 0.16
- 1% percentile: -0.8501208877563475
- 99% percentile: 1.371586303710937

No sources have both WFC z (aperture) and RCS z (aperture) values.
RCS z (total) - WFC z (total):
- Median: -0.07
- Median Absolute Deviation: 0.22
- 1% percentile: -1.0420454406738282
- 99% percentile: 1.9726009368896484

GPC1 z (aperture) - CFHT Megacam z (aperture):
- Median: -0.06
- Median Absolute Deviation: 0.19
- 1% percentile: -3.8111167526245113
- 99% percentile: 1.116339416503906

GPC1 z (total) - CFHT Megacam z (total):
- Median: 0.15
- Median Absolute Deviation: 0.10
- 1% percentile: -2.2986411094665526
- 99% percentile: 0.8422527313232419

No sources have both CFHT Megacam z (aperture) and RCS z (aperture) values.
RCS z (total) - CFHT Megacam z (total):
- Median: -0.05
- Median Absolute Deviation: 0.24
- 1% percentile: -2.377850933074951
- 99% percentile: 1.2616420936584478

No sources have both GPC1 z (aperture) and RCS z (aperture) values.
RCS z (total) - GPC1 z (total):
- Median: -0.18
- Median Absolute Deviation: 0.10
- 1% percentile: -0.9303009223937988
- 99% percentile: 0.5799712562561041

No sources have both GPC1 y (aperture) and RCS y (aperture) values.
No sources have both GPC1 y (total) and RCS y (total) values.

III - Comparing magnitudes to reference bands

Cross-match the master list to SDSS magnitudes.

master_catalogue_coords = SkyCoord(master_catalogue['ra'], master_catalogue['dec'])

III.a - Comparing u, g, r, i, and z bands to SDSS

The catalogue is cross-matched to SDSS-DR13 withing 0.2 arcsecond.

We compare the u, g, r, i, and z magnitudes to those from SDSS using fiberMag for the aperture magnitude and petroMag for the total magnitude.

sdss = Table.read("../../dmu0/dmu0_SDSS-DR13/data/SDSS-DR13_ELAIS-N2.fits")
sdss_coords = SkyCoord(sdss['ra'] * u.deg, sdss['dec'] * u.deg)

idx, d2d, _ = sdss_coords.match_to_catalog_sky(master_catalogue_coords)
mask = (d2d < 0.2 * u.arcsec)

sdss = sdss[mask]
ml_sdss_idx = idx[mask]

for band_of_a_kind in [u_bands, g_bands, r_bands, i_bands, z_bands]:
    for band in band_of_a_kind:
        
        sdss_mag_ap = sdss["fiberMag_{}".format(band[-1])]
        master_cat_mag_ap = master_catalogue["m_ap_{}".format(band.replace(" ", "_").lower())][ml_sdss_idx]
    
        nb_compare_mags(sdss_mag_ap, master_cat_mag_ap,
                        labels=("SDSS {} (fiberMag)".format(band[-1]), "{} (aperture)".format(band)))
    
        sdss_mag_tot = sdss["petroMag_{}".format(band[-1])]
        master_cat_mag_tot = master_catalogue["m_ap_{}".format(band.replace(" ", "_").lower())][ml_sdss_idx]
        
        nb_compare_mags(sdss_mag_ap, master_cat_mag_ap,
                        labels=("SDSS {} (petroMag)".format(band[-1]), "{} (total)".format(band)))

WFC u (aperture) - SDSS u (fiberMag):
- Median: -0.30
- Median Absolute Deviation: 0.18
- 1% percentile: -1.6183706665039062
- 99% percentile: 0.9621364212036125

WFC u (total) - SDSS u (petroMag):
- Median: -0.30
- Median Absolute Deviation: 0.18
- 1% percentile: -1.6183706665039062
- 99% percentile: 0.9621364212036125

CFHT Megacam u (aperture) - SDSS u (fiberMag):
- Median: -0.04
- Median Absolute Deviation: 0.58
- 1% percentile: -1.6583412170410157
- 99% percentile: 3.297002315521247

CFHT Megacam u (total) - SDSS u (petroMag):
- Median: -0.04
- Median Absolute Deviation: 0.58
- 1% percentile: -1.6583412170410157
- 99% percentile: 3.297002315521247

WFC g (aperture) - SDSS g (fiberMag):
- Median: -0.44
- Median Absolute Deviation: 0.11
- 1% percentile: -1.1331189155578614
- 99% percentile: 0.5049597740173323

WFC g (total) - SDSS g (petroMag):
- Median: -0.44
- Median Absolute Deviation: 0.11
- 1% percentile: -1.1331189155578614
- 99% percentile: 0.5049597740173323

CFHT Megacam g (aperture) - SDSS g (fiberMag):
- Median: -0.36
- Median Absolute Deviation: 0.18
- 1% percentile: -1.1016152000427246
- 99% percentile: 3.0158825111388934

CFHT Megacam g (total) - SDSS g (petroMag):
- Median: -0.36
- Median Absolute Deviation: 0.18
- 1% percentile: -1.1016152000427246
- 99% percentile: 3.0158825111388934

GPC1 g (aperture) - SDSS g (fiberMag):
- Median: -0.48
- Median Absolute Deviation: 0.19
- 1% percentile: -1.7155875587463378
- 99% percentile: 1.8861633110046394

GPC1 g (total) - SDSS g (petroMag):
- Median: -0.48
- Median Absolute Deviation: 0.19
- 1% percentile: -1.7155875587463378
- 99% percentile: 1.8861633110046394

No sources have both SDSS g (fiberMag) and RCS g (aperture) values.
No sources have both SDSS g (petroMag) and RCS g (total) values.
WFC r (aperture) - SDSS r (fiberMag):
- Median: -0.34
- Median Absolute Deviation: 0.07
- 1% percentile: -0.890346794128418
- 99% percentile: 0.36871064186096253

WFC r (total) - SDSS r (petroMag):
- Median: -0.34
- Median Absolute Deviation: 0.07
- 1% percentile: -0.890346794128418
- 99% percentile: 0.36871064186096253

CFHT Megacam r (aperture) - SDSS r (fiberMag):
- Median: -0.27
- Median Absolute Deviation: 0.13
- 1% percentile: -0.8717788696289063
- 99% percentile: 3.9546873378753666

CFHT Megacam r (total) - SDSS r (petroMag):
- Median: -0.27
- Median Absolute Deviation: 0.13
- 1% percentile: -0.8717788696289063
- 99% percentile: 3.9546873378753666

GPC1 r (aperture) - SDSS r (fiberMag):
- Median: -0.37
- Median Absolute Deviation: 0.11
- 1% percentile: -1.1465348052978515
- 99% percentile: 0.8678107452392576

GPC1 r (total) - SDSS r (petroMag):
- Median: -0.37
- Median Absolute Deviation: 0.11
- 1% percentile: -1.1465348052978515
- 99% percentile: 0.8678107452392576

No sources have both SDSS r (fiberMag) and RCS r (aperture) values.
No sources have both SDSS r (petroMag) and RCS r (total) values.
WFC i (aperture) - SDSS i (fiberMag):
- Median: -0.39
- Median Absolute Deviation: 0.08
- 1% percentile: -0.9288136291503907
- 99% percentile: 0.4154920196533197

WFC i (total) - SDSS i (petroMag):
- Median: -0.39
- Median Absolute Deviation: 0.08
- 1% percentile: -0.9288136291503907
- 99% percentile: 0.4154920196533197

GPC1 i (aperture) - SDSS i (fiberMag):
- Median: -0.36
- Median Absolute Deviation: 0.09
- 1% percentile: -0.9859428405761719
- 99% percentile: 0.43151168823242997

GPC1 i (total) - SDSS i (petroMag):
- Median: -0.36
- Median Absolute Deviation: 0.09
- 1% percentile: -0.9859428405761719
- 99% percentile: 0.43151168823242997

No sources have both SDSS i (fiberMag) and RCS i (aperture) values.
No sources have both SDSS i (petroMag) and RCS i (total) values.
WFC z (aperture) - SDSS z (fiberMag):
- Median: -0.33
- Median Absolute Deviation: 0.11
- 1% percentile: -1.2395364761352539
- 99% percentile: 0.7405158233642565

WFC z (total) - SDSS z (petroMag):
- Median: -0.33
- Median Absolute Deviation: 0.11
- 1% percentile: -1.2395364761352539
- 99% percentile: 0.7405158233642565

CFHT Megacam z (aperture) - SDSS z (fiberMag):
- Median: -0.19
- Median Absolute Deviation: 0.19
- 1% percentile: -1.0708820343017578
- 99% percentile: 3.6499409675597967

CFHT Megacam z (total) - SDSS z (petroMag):
- Median: -0.19
- Median Absolute Deviation: 0.19
- 1% percentile: -1.0708820343017578
- 99% percentile: 3.6499409675597967

GPC1 z (aperture) - SDSS z (fiberMag):
- Median: -0.29
- Median Absolute Deviation: 0.13
- 1% percentile: -1.1712089157104493
- 99% percentile: 1.001730842590327

GPC1 z (total) - SDSS z (petroMag):
- Median: -0.29
- Median Absolute Deviation: 0.13
- 1% percentile: -1.1712089157104493
- 99% percentile: 1.001730842590327

No sources have both SDSS z (fiberMag) and RCS z (aperture) values.
No sources have both SDSS z (petroMag) and RCS z (total) values.

Keeping only sources with good signal to noise ratio

From here, we are only comparing sources with a signal to noise ratio above 3, i.e. roughly we a magnitude error below 0.3.

To make it easier, we are setting to NaN in the catalogue the magnitudes associated with an error above 0.3 so we can't use these magnitudes after the next cell.

for error_column in [_ for _ in master_catalogue.colnames if _.startswith('merr_')]:
    column = error_column.replace("merr", "m")
    keep_mask = np.isfinite(master_catalogue[error_column])
    keep_mask[keep_mask] &= master_catalogue[keep_mask][error_column] <= 0.3
    master_catalogue[column][~keep_mask] = np.nan

IV - Comparing aperture magnitudes to total ones.

nb_ccplots(
    master_catalogue['m_cfht_megacam_r'],
    master_catalogue['m_ap_cfht_megacam_r'] - master_catalogue['m_cfht_megacam_r'],
    "r total magnitude (CFHT)", "r aperture mag - total mag (CFHT)",
    master_catalogue["stellarity"],
    invert_x=True
)

Number of source used: 764852 / 1783223 (42.89%)

V - Color-color and magnitude-color plots

nb_ccplots(
    master_catalogue['m_wfc_i'] - master_catalogue['m_swire_irac1'],
    master_catalogue['m_wfc_g'] - master_catalogue['m_wfc_i'],
    "WFC i - IRAC1", "g - i (WFC)",
    master_catalogue["stellarity"]
)

Number of source used: 102035 / 1783223 (5.72%)

nb_ccplots(
    master_catalogue['m_cfht_megacam_u'] - master_catalogue['m_cfht_megacam_g'],
    master_catalogue['m_cfht_megacam_g'] - master_catalogue['m_cfht_megacam_r'],
    "u - g (CFHT)", "g - r (CFHT)",
    master_catalogue["stellarity"]
)

Number of source used: 665469 / 1783223 (37.32%)

nb_ccplots(
    master_catalogue['m_irac3'] - master_catalogue['m_irac4'],
    master_catalogue['m_swire_irac1'] - master_catalogue['m_swire_irac2'],
    "IRAC3 - IRAC4", "IRAC1 - IRAC2",
    master_catalogue["stellarity"]
)

Number of source used: 19548 / 1783223 (1.10%)